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FOREWORD 


This document attempts to illustrate the application of cost- 
effectiveness analysis as a practical tool for the systems designer. 
In particular, a method for selecting which parameters should be 
monitored in order to assess the operational status of a telemetry 
system is illustrated. 

A paper, "Cost-Effectiveness Analysis: An Appreciation, " by 
E. S. Quade of the Rand Corporation is presented as an appendix 
to this document. As the primary purpose of this document is to 
illustrate the use of cost-effectiveness analysis, it is recommended 
that this appendix be read prior to reading the main document. 
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DESIGN OF A STATUS MONITOR 
USING COST-EFFECTIVENESS ANALYSIS 


1. INTRODUCTION 

Figure 1 shows the general design technique used to design a status moni- 
tor* for a satellite ground station. This document will describe in detail how 
each step of the design technique was accomplished to produce the design of a 
status monitor for the telemetry portion of a ground station. 

The telemetry installation considered in this document was proposed for 
the new Network Test and Training Facility (NTTF) located at the Goddard 
Space Flight Center in Greenbelt, Maryland. Figure 2 is the basic functional 
diagram of this installation. 


2. THE STATUS MONITOR DESIGN 

The primary function which must be performed when designing a status 
monitor is to make a decision as to which, of all the possible parameters that 
could be monitored, should be monitored. The basis for making this selection 
is to determine which parameters are the most effective in assessing the status 
of the telemetry installation and are the least costly to monitor. 

The effectiveness of assessing the status and the cost to do so are, in gen- 
eral, opposing factors. The selection thus becomes one of determining the 
monitoring configuration which exhibits the best effectiveness to cost ratio 
(effectiveness/cost) . 

This section of the document will describe how each step of the design was 
accomplished in order to determine the optimum status monitor. 

2 . 1 Step 1 - Define the Objective of the Status Monitor 

This step is the most important step in the design process. Designing a 
system which meets the wrong objectives is like solving the wrong problem. 

The resulting design may be completely inadequate or, as a minimum, a 
suboptimum design will result. The most frequent error is selecting objectives 
which are too narrow in scope. 


*The term "status monitor" as used in this paper refers to the tests to be made in order to 
assess the operational ability of the station and the hardware used to implement the tests. 


1 




(6) Screen the alternatives to determine the monitoring configuration 
with the greatest effectiveness/cost rating. 


Figure 1 . A General Design Technique for a 
Satellite Ground Station Status Monitor 
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Figure 2. GSFC Telemetry Basic Functional Layout 



















The objective of monitoring the status of a ground station is to increase the 
ability of the station to meet its operational requirements. The next step in de- 
signing the status monitor is therefore to describe the operational requirements 
of the station. This step is described in Section 2.2. 

The main operations which a status monitor may perform to meet the de- 
fined objective are as follows: 

(1) Failure* determination - This operation is to determine if the station 
is capable of operating within prescribed tolerances. This includes determining 
if all interfaces between sybsystems are properly configured and that the equip- 
ments are capable of performing their required function. This operation must 
be performed before corrective action can be taken. The corrective action may 
take the form of repairing the malfunction, replacing the faulty unit with a good 
one, or notifying operation-control that the station cannot meet the mission re- 
quirement, (then operation-control could take some corrective action like as- 
signing the function to a back-up station). No matter which action is taken the 
probability of meeting the mission requirements will be increased. 

(2) Failure isolation - In case of an indicated failure, information con- 
cerning the location of the faulty item is provided. This operation has the effect 
of decreasing the down-time required to correct the indicated failure, again in- 
creasing the probability of meeting the mission requirements. 

(3) Failure prediction - This is the operation of recognizing incipient 
failures in the system. This operation is accomplished by determining parame- 
ter drift and drift rates. This operation has the effect of decreasing the number 
of failures thus also, increasing the probability of meeting the mission 
requirements . 

2.2 Step 2 - Describe the Operational Requirements of the Ground Station 

Operational requirements are those functions which the ground station must 
perform to accomplish all its mission objectives. 

The complete operation of the NTTF is shown in Figure 3. As illustrated, 
the station operation has been separated into five functions (tracking, telemetry, 
command, timing, and monitoring and maintenance). Figure 3 has also been 
divided into the various modes in which the station operates. These modes 
(availability, prepass readiness, spacecraft (S/C) acquisition, telemetry (T/M) 


♦Failure is defined here to mean any malfunction which prevents the system from meeting 
its total mission requirements. 
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assessment, etc.), are indicated at the very top of Figure 3. The operational 
requirements necessary for the station to perform each mode are indicated as 
rectangles on Figure 3. 

The information needed to construct Figure 3 was obtained by a review of 
the STADAN Orientation Manual, GSFC Operations Plans, and on-site 
procedures. 

This step, of describing on paper the operational requirements, is quite 
often left out of the intuitive design approach. When dealing with systems this 
step becomes necessary as it provides several important "bookkeeping" 
functions. 

By setting down operational requirements on paper there is less chance of 
omitting any requirements. This step also enables recognition of interfaces 
between the various operations (e.g. while the status monitor design being con- 
sidered is only for the telemetry portion of the station, Figure 3 indicates 
there exists interfaces which must be considered with the command and timing 
operations). Also indicated in Figure 3, represented as circles, is the input/ 
output functions of the station (e.g. station reporting requirements and source 
of operating requirements) . 

The accomplishment of this step provides the basis for specifying perform- 
ance indicators, Step 3. 

2.3 Step 3 - Specify Measurable Quantities Which Most Directly Repre- 
sent the Operational Requirements (performance indicators) 

For each operational requirement, represented as rectangles in Figure 3, 
measurable quantities which most directly represent the requirements are de- 
rived. These measurable quantities are called "performance indicators. " 

Figure 4 illustrates the derivation of some performance indicators from 
the operational requirements. As illustrated, sometimes the requirements 
must be broken into sub-requirements before a measurable indicator of per- 
formance is derived. Thus there will be one or more performance indicators 
for each operational requirement. 

The performance indicators are determined by reviewing the operational 
requirements (from Figure 3) and the hardware used in the system to accom- 
plish the operational requirements (from equipment manuals) . 
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PERFORMANCE f DETECTION PHASE-LOCK LOOP RECEIVER LOCK FRE ®^ CY ERROR RATE CORRESPONDENCE 
INDICATORS THRESHOLD LOCKED LIGHT ™ (I0- 3 error%/ WITH FORMAT 

(tolerance) (threshold ±2db) (loop threshold ±2db) (threshold ±2db) l bit) (mission dependent) 

L bandwidth) 


With the accomplishment of this step the designer has specified the mea- 
surable quantities which indicate that the station is able to meet its operational 
requirements. 

The next step in this design process is to develop methods of measuring 
the performance indicators or measuring other parameters which are related 
to the performance indicators. 

2.4 Step 4 - Develop Methods of Measuring the Performance Indicators 
and, As An Alternate, Parameters Related to the Perform- 
ance Indicators 

Performance indicators can be measured directly or a measurement of 
some other quantity which relates to the performance indicators can be made. 
Some measurement configuration may be quite effective in its ability to deter- 
mine the status of the performance indicators. Alternatively, other configura- 
tions may not be as effective but may be considerably cheaper to implement. 

The designer is thus faced with the age-old tradeoff problem of determining how 
much performance should he buy. 

This step requires "good hard-headed engineering" and a detailed analysis 
of the equipment and operations. Here again the designer needs some way of 
keeping track of the individual detailed measurements while maintaining a com- 
plete overall picture of the alternate design configurations. The method of 
performing this bookkeeping is through the development of a measurement net- 
work diagram together with a table describing the measurement details. 

An example of a trivial measurement network is illustrated in Figure 5. 

On Figure 5 measurements 1 and 2 represent two performance indicators. Al- 
ternate to measurement 1 measurements 3 and 4 could be made. Measurements 
6 and 4 is another alternative to measurement 1. Also illustrated is a case 
where one measurement (9) is an alternative to measuring performance indica- 
tors 1 and 2 . 

Corresponding to each measurement in the measurement network is an 
entry in a table which describes how and where the measurement is performed. 

The next step in solving the tradeoff problem is to specify appropriate 
measures of effectiveness and cost with which to make the evaluation. 
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Figure 5. Example of a Measurement Network 

2. 5 Step 5 - Specify Measures of Effectiveness and Cost to be Used in 

Rating the Methods of Measuring the Performance Indicators 
and Their Alternates 

The selection of the appropriate measures of effectiveness and cost is 
largely a matter of intuition (otherwise known as good hard-headed engineering) . 
Here again, by putting these factors down on paper one is made aware of the 
reasons why a particular configuration was selected over others. 

The measures of effectiveness (or effectiveness factors) for a status moni- 
tor are: 

(a) How well the parameters monitored reflect variation of the perform- 
ance indicators (these are called sensitivity factors). 

(b) How well the parameters monitored recognize those items which ex- 
hibit a relatively high failure rate (these are called reliability factors) . 

(c) Are the parameters which are most important to the overall mission 
success monitored (these are called dependency factors) . 

(d) The amount of uncertainty about the status of the station that can be 
resolved in the least amount of testing time (these are called time 
factors) . 

The measures of costs (or cost factors) for a status monitor are: 

(a) Cost of equipment procurement; 
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(b) Cost of implementing the status monitor system; 

(c) Cost of operating and maintaining the status monitoring system; 

(d) Cost of data gathering and display of the status information. 

If some measures of effectiveness (or cost) are more important than others, 
relative weightings can be assigned to the different effectiveness (cost) factors 
to reflect this in the tradeoff analysis. 

The methods for arriving at the various effectiveness factor ratings are as 
follows: 

(a) Sensitivity factors - The theoretical relation between the measured 
parameter and the related performance indicator is used to estimate 
the sensitivity rating. This theoretical relationship may exist in the 
form of an equation, graph, table or it may need to be arrived at 
empirically. 

(b) Reliability factors - Reliability ratings can be obtained from failure 
data, or if non-existent, from a reliability analysis using a parts 
count or a monte-carlo simulation, etc. 

(c) Dependency factors - Dependency ratings are obtained from an analysis 
of the station configuration and the operational requirements. 

(d) Time factors - Ratings of time factors are the result of test set-up 
time, testing time, and status interpretation time. Methods for esti- 
mating these time factors may require performing some empirical 
tests and possibly making a monte-carlo simulation. 

The methods for arriving at the cost factor ratings are as follows:* 

(a) Cost of equipment procurement - This rating is obtained from a sur- 
vey of instrumentation costs and is the sum of the costs of the different 
test functions** used. 

(b) Cost of implementation - A reasonable estimate of relative implemen- 
tation cost appears to be simply the sum of the number of different 
test functions used. 

*The reasoning behind these methods for arriving at cost factor ratings are presented in 
"A Study of Station Performance Criteria", by Operations Reasearch, Inc., under Contract 
NAS5-9910 for NASA-Goddard Space Flight Center. 

**A test function is one particular measurement. 
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(c) Cost of operation and maintenance - The relative cost of operation can 
be estimated from the total number of test functions. The relative 
cost of maintenance is estimated by the number of different test 
functions. 

(d) Cost of data gathering and display - A relative rating for this cost fac- 
tor can be estimated from the total number of test functions. 

For each measurement path in the network the effectiveness and cost rating 
of the measurement must be determined. There is one effectiveness rating for 
each measurement path. This rating is the sum of the ratings of the individual 
weighted effectiveness factors. There are two cost ratings for each measure- 
ment path. One cost rating is the sum of the ratings of the individual cost fac- 
tors and is called the independent cost rating. The second cost rating takes 
into account the possibility of using the same instrumentation to make several 
measurements. This dependent cost factor is equal to the sum of all the cost 
factors less the cost of procurement of duplicate equipment. 

2. 6 Step 6 - Screen the Alternatives to Determine the Monitoring Con- 
figuration With the Greatest Effectiveness /Cost Rating 

In this step the measurement network is analyzed to determine the set of 
measurements which exhibits the highest effectiveness/cost ratio. 

If the measurement network is small, the selection could be made manually 
using such methods as matrix analysis or graph theory. If the measurement 
network becomes large the selection must utilize computer analysis. 

The measurement network is a direct analog to the PERT (Program Evalu- 
ation and Review Technique) network. Thus, a standard PERT critical path 
program can be used. Part I of the PERT computer program calculates the 
"best" path as a free running program (that is without constraints on effective- 
ness or cost) . If it is desired to place constraints on either minimum effec- 
tiveness or maximum cost, or both, the PERT-Part II (PERT with constraints) 
program is used. 

In this program the computer goes through all possible paths, keeping 
track of both dependent and independent cost factors, and prints out the path 
which exhibits the highest effectiveness/cost ratio. This path makes up the 
best of the alternate sets of tests for monitoring the status of the telemetry por- 
tion of the NTTF. 
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Appendix A is a description of the optimum set of tests, as determined by 
the cost-effectiveness analysis, for monitoring the status of the telemetry por- 
tion of the NTTF. This description is presented to give the reader who is 
familiar with the station operation an indication of how completely the tests 
monitor the status of the telemetry installation. 


3. CONCLUSION 

This design procedure provides all the advantages of an intuitive design 
since the steps taken in this cost-effectiveness design includes exactly the same 
steps taken in an intuitive design. However, by performing this more rigorous 
design procedure the following significant advantages are achieved: 

(a) By putting down on paper the effectiveness and cost factors, the rea- 
sons for selecting a particular system design is made in terms mean- 
ingful to everyone; 

(b) By setting down on paper the station operational requirements and the 
performance indicators the determination of what should be displayed 
to the operator can be easily and rationally made. 

(c) By being forced to specify both the overall picture and the fine details, 
the designer can very easily determine how changes in the station con- 
figuration (or operation) affect the status monitor design. This fact 
will enable the status monitor to be efficiently modified to accommo- 
date the station changes; 

(d) By requiring that factors of sensitivity, reliability, time and cost be 
determined, the designer is made aware of any lack of available data 
needed to determine the various ratings. The designer then knows 
what experiments need to be made before a status monitor design can 
be confidently selected or what data should be reported and in what 
format, to easily provide for future generation status monitoring 
systems. 
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APPENDIX A 


DESCRIPTION OF THE SELECTED 
STATUS MONITOR TESTS 
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STATUS MONITORING TESTS 


Test Group A: Frequency Modulation (FM) Mission Tests. 


1. Improvement Threshold Test. 

This test is to determine the improvement threshold point as given by 
KTB + NF + lOdb (where KTB is the received noise power and NF is the noise 
figure inherent to the system) . The improvement threshold is defined as the 
point on the Signal-to-Noise Ratio (SNR)/Carrier-to-Noise Ratio (CNR) curve 
where the output SNR increases db-for-db with the input CNR. This test per- 
mits the determination of any degradations in the bandwidth (B) or the noise 
figure (NF) of the basic telemetry receiving system. 

2. Combiner Performance Test. 

This test is to determine whether the CNR contributed by each re- 
ceived channel, at the improvement threshold point, is equal and that the two 
receiver outputs are being combined properly. 

3. Non-Linear Noise Measurement Test. 

This test is to determine, at the combiner output, the total phase non- 
Linearities that may occur within the receiving system. 

4. Automatic Frequency Control (AFC) Loop Dynamic Response Test. 

The combiner output is monitored to determine that the AFC loop dyna- 
mic response is capable of handling the doppler shifted transmission from any 
spacecraft. 


Test Group B: Phase Modulation (PM) Mission Tests. 


1. Acquisition Threshold Test. 

This test will show system sensitivity and variations in the phase-lock 
loop bandwidth by determining the carrier power required to achieve lock and 
to maintain lock. 
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2. Combiner Performance Test. 


(a) This test will be to determine the improvement, over single re- 
ceiver operation, provided by the combiner over the range of lock threshold to 
saturation, and 

(b) To determine that the combiner switches properly at low CNR's. 

3. Doppler Extracting Test. 

This test will be to determine the ability of the Electrac Phase-locked 
Demodulator to extract and record the doppler phase shift. 

4. Non-Linear Noise Measurement Test. 

This test will be to determine, at the combiner output, the total phase 
non-linearities that may occur within the receiving system. 


Test Group C: Baseband Equipment Validation Tests. 


1. Pulse Code Modulation (PCM) Validation Tests. 

A test at the output of the (PCM/DHE) to determine the PCM error 
rate vs CNR, at a given confidence limit, through the complete receiving sys- 
tem, tape recorders, and PCM/DHE. This test is applicable to all Time Di- 
vision Multiplex signals. 

2. Pulse Frequency Modulation (PFM) Validation Test. 

A check at the chart recorder output to determine the saturation, 
noise and non-linearities of the PFM system performance. (The PFM system 
includes telemetry receiver, combiner, subcarrier multiplexer /demultiplexer, 
and chart recorders.) This test is applicable to all Frequency Division Multi- 
plex signals. 

3. Antenna and Servo Performance Test. 

To determine the correct electrical and mechanical operation of the 
antenna and servo tracking loop. (The antenna, tracking receiver and servo 
equipment are checked. ) 
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COST-EFFECTIVENESS ANALYSIS: AN APPRECIATION 


* 

E. S, Quade 
The RAND Corporation 
Santa Monica, California 


This paper attempts to clarify the nature and 
scope of cost-effectiveness analysis and to point 
out its proper role as an aid to decisionmaking. 

It discusses, in the context of national security 
problems, the reliability and limitations of such 
analyses and ways to improve its quality. 

Introduction 

Cost-effectiveness is not a catchword to sug- 
gest we are doing something new, for the need to 
consider cost in relation to effectiveness must 
have occurred to the earliest planners. What is 
novel, however, is the marvelous refinement of 
methods for relating cost to performance that has 
taken place in the last few years and the accep- 
tance of these methods at high policy levels where 
they are often proposed as a panacea for all the 
ills of intricate decisionmaking. 

PefinlU<?ns 

What is a cost-effectiveness analysis? 

Broadly defined (too broadly for my taste) it Is 
any analytic study designed to assist a decision- 
maker Identify a preferred choice from among pos- 
sible alternatives. In a military context, typi- 
cal analyses might tackle such questions as the 
extent to which aircraft should be repaired at a 
depot rather than on the base; the possible char- 
acteristics of a new strategic bomber and whether 
one should be developed or not; whether tactical 
air wings or carrier task forces should be sub- 
stituted for U. S. ground divisions in Europe; or 
whether we should modify the test ban treaty now 
that the Chinese Communists have nuclear weapons 
and, if so, how. One stage of each such analysis 
involves a comparison of alternative courses of 
action in terms of their costs and their effec- 
tiveness in attaining some specific objective. 

This is cost-effectiveness analysis, narrowly 
defined. Usually this comparison takes the form 
of an attempt to minimize the cost implications 
subject to some mission requirement (which in 
broad problems is not likely to be measurable in 
dollar terms) or, conversely, to maximize some 
physical measure of performance subject to a 
budget constraint. 

Since such comparisons receive the lion's 
share of attention by the participants, the entire 
study is often called a cost-effectiveness analy- 
sis. But this name emphasizes Just one aspect of 


Any views expressed in tills paper are those 
of the author. They should not be interpreted as 
reflecting the views uf The RAND Corporation or 
the official opinion or policy ol any of its gov- 
ernmental or private research sponsors. Papers 
are reproduced by The RAND Corporation as a 
courtesy to members of its staff. 


the study. For advice on broad questions of policy 
such as those related to national defense (where 
cost-effectiveness has been most extensively used), 
facets of the problem other than the comparison of 
alternatives may be of great significance. Among 
these are: the specification of sensible objec- 

tives, the determination of a satisfactory way to 
measure performance, the influence of consider- 
ations that can't be quantified, or the discovery 
of better alternatives. 

Let me try to illustrate this last point with 
a homely example. 

Suppose a family has decided to buy a tele- 
vision set. Not only is their objective fairly 
clear, but, if they have paid due attention to the 
advertisements, their alternatives are well- 
defined. The situation is then one for cost-oifec- 
tlveness analysis. The only significant questions 
the family need answer concern the differences 
among the available sets in performance and cost. 
With a little care, making proper allowance for 
financing, depreciation, and maintenance , they can 
estimate, say, the five year procurement and 
operating cost of any particular set and do so 
with a feeling that they are well inside the ball 
park. They will discover, of course, that finding 
a standard for measuring the performance of the 
various sets is somewhat more difficult. For one 
thing, it may have many aspects--co lor quality, 
the option for remote control, portability, screen 
size, and so forth. But, ordinarily, one consider- 
ation — perhaps color--determines a price class. 

On this basis, one can look at color sets, compare 
costs against color quality, and determine a best 
buy. 

Now suppose the family finds they have more 
money to spend and thus decide to increase their 
standard of living — a decision similar to one to 
strengthen the U.S, defense posture by increasing 
the military budget. This is a situation calling 
for a broader analysis. They first need to 
investigate their goals or objectives and look 
into the full range of a lternat ives--a new car, 
a piano, a trip to Europe. They then must find 
ways to measure how well these alternatives 
accomplish their goals and establish criteria for 
choice among them. Because the alternatives are 
so dissimilar, determining what they want to do is 
the major problem; how to do it and how to deter- 
mine what it costs is a comparatively minor one. 

In brief, to handle a broad prob lem adequate l y 
a study must look at the entire problem and look 
at it in its proper context. Characteristically, 
such an analysis will involve a systematic inves- 
tigation of the decisionmaker's objectives and of 
the relevant criteria; a comparison--quant i tat i ve 
where possible— of the costs, effectiveness, risks, 
and timing associated with the alternative pel w it* 
or strategies for achieving each objective. and .n 
attempt to formulate better a It et nat i ves il those 
examined are found wanting. Although l pieiei lh. 
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name "systems analysis" for the broader analysis, 
in what follows I'll use the term cost-effective- 
ness for the full range, broad and narrow, of 
analytic approaches to aid a decisionmaker with 
problems of choice. 

The Essence of the Method 

What is there about an analytic approach that 
makes it better or more useful than other ways to 
furnish advice--than , say, an expert or a commit- 
tee? In areas such as defense planning, where 
there is no accepted theoretical foundation, 
advice obtained from experts working individually 
or as a committee depends iargely on judgment and 
intuition. So does the advice from cost-effec - 
t iveness analysis. But the virtue of analysis is 
that it permits the judgment and int^tinn nf 
experts in” many nelds to be combined systemati- 
cally and efficiently to yield results that can 
transcend those of any individual or conmittee. 

The essence of the method is to construct and 
operate within a 'Vnodel"--a simplified, stylized 
representation of the real world which abstracts 
the cause and effect relationships essential to 
the question being studied. Such a model--which 
may take such varied forms as a set of mathemati- 
cal equations or a computer program, a war game, 
or even a purely verbal scenario--introduces a 
precise structure and terminology that serve 
primarily as a means of conmunicat ion , enabling 
the participants in the study to make their 
judgments in a concrete context and with proper 
reference to the judgment of others. Moreover, 
through fecdback--the results of computation, the 
countermoves in the war game, or the critique of 
the scenario--the model helps the experts to 
revise their earlier judgments and thus to arrive 
at a clearer understanding of the problem and its 
context . 

The central importance of the model (or the 
models, for it may be inappropriate or absurd to 
attempt to incorporate all aspects of a problem in 
a single formulation) can be seen most readily, 
perhaps, by looking at its role in the choice of 
o I tornat ives. 

Having formulated and researched the problem 
- - that is, clarified the issues, limited the 
extent of the inquiry, searched out the necessary 
data and relationships, discovered what objectives 
the decisionmaker is, or should be, trying to 
attain, and how to measure the extent to which 
they are, in fact, attained, and built various 
model s--the process is somewhat as follows. (See 
chart.) To begin, the various alternatives or 
means by which one can hope to attain the objec- 
tives (which may have to be discovered or invented 
as part of the analysis) are examined by means of 
the mode 1s . These models tell us what we can 
expect from each particular alternative with 
respect to such things as attrition, reliability, 
and so forth, and what the costs are. The 
measure of effectiveness then tells us the extent 
to which each objective is attained. A criterion 
or rule of choice can then be vised to weigh the 
costs against performance and thus arrange the 
alternatives in order of preference. 

This process may be difficult to carry out. 


For instance, consider the estimation of total 
system reliability. Often this is represented by 
the mean time between failures (MTBF) , calculated 
by taking the reciprocal of the sum of the recip- 
rocals of the subsystem MTBF's. The exponential 
distribution is then used to obtain the probabil- 
ity that no system failure will occur in a time 
period. This simple scheme involves at least four 
tacit assumptions: 

o The time between failures is exponen- 
tially distributed, 

o Failures of subsystems are independent, 

o A subsystem failure implies a system 
failure , 

o Subsystems are utilized equally in time. 

Ideally, the equations which express reliability 
should account for subsystem failure rates, 
redundancies, dependencies, and utilization. 

While complicated, this is not beyond the capa- 
bilities of a computer. But the estimates of the 
subsystem failure rates themselves depend on 
partial measurements and intuitive judgments of 
the influence of temperature, humidity, dust, 
shock, stress, vibration, operating cycle, and the 
environment. The end result may be that predic- 
tions from the reliability model are highly uncer- 
tain. 

In fact, things are seldom tidy. Too often 
alternatives are not adequate to attain the objec- 
tives; measures of effectiveness do not really 
measure the extent to which the objectives are 
attained; the predictions from other models, as 
well as from the reliability model, are full of 
uncertainties; and other criteria which look 
almost as attractive as the one chosen may lead 
to a different order of preference. When this 
happens, no one is happy and we must take another 
approach. Dissent and discussion force modifica- 
tion of original ideas about objectives and alter- 
natives are redesigned. The key to successful 
analysis is a continuous cycle of formulating the 
problem, selecting the objectives, designing 
better alternatives, collecting data, building new 
models, weighing cost against performance, ques- 
tioning assumptions and data, reexamining the 
objectives, opening new alternatives, and so on 
until satisfaction is obtained or time or money 
forces a cutoff. 

The Limitations 

Analysis of this type is not only difficult 
to do velL but even when well done there are many 
limitations. Some of these are due to limitations 
inherent in all analysis of choice. Others are 
due to the difficulties encountered in copin ft with 
such things as the varying times at which alter- 
natives become available or uncertainty about the 
enemy. Still others are flaws or errors which, 
hopefully, will disappear as we learn to do better 
and more thorough analyses. The most dangerous 
source of defects, however, is an attention bias . 
It is frequently caused by the cherished beliefs 
or unconscious adherence to a "party line" that 
all organizations foster to some extent. 


25 



It is important to remember that all analysis 
of choice falls short of scientific research. No 
matter how we strive to maintain standards of 
scientific inquiry or how closely we attempt to 
follow scientific methods , we cannot turn cost- 
effectiveness analysis into science. Its objec- 
tive, in contrast to that of science, is primarily 
to recommend --or at least to suggest--policy , 
rather than merely to understand and predict. 

Like engineering, it seeks to use the results of 
science to do things well and cheaply. Yet it 
differs from ordinary engineering in its enormous 
responsibility, in sometimes being forced by the 
nature or urgency of a problem to substitute 
intuition for verifiable knowledge, in the unusual 
difficulty of apprais ing--or even disco ver ing--a 
value system applicable to its problems, and in 
the absence of ways to test its validity. 

Except for this inability to verify, cost- 
effectiveness analysis may still look like a 
purely rational approach to decisionmaking, a 
coldly objective, scientific method free of pre- 
conceived ideas and partisan bias and judgment and 
intuition. But it isn't really. Human judgment 
is used in designing the analysis: in deciding 

what alternatives to consider, what factors are 
relevant, what the interrelations between these 
factors are, and what numerical values to choose. 
Moreover, It is human judgment which analyzes and 
interprets the results of the analysis. This fact 
--that judgment and intuition permeate all analy- 
sis--should be remembered when we examine the 
results that come, with apparent high precision, 
from analysis. 

But it is the inherent limitations of the 
analysis, not errors, that confine it to an 
advisory role. I shall single out three of them 
for further comment: analysis is necessarily 

incomplete; measures of effectiveness are inevi- 
tably approximate; and ways to predict the future 
are lacking. 

Analysis is necessarily incomplete 

Time and money costs obviously place sharp 
limits on how far any inquiry can be carried. 

Other costs are important here too. For instance, 
we would like to find out what the Chicoms would 
do if we put an end to all military aid to South- 
east Asia. One way to get t;his information would 
be to stop such aid. But while the immediate 
dollar cost would be low, the likelihood of other 
costs occurring in time precludes at once this 
type of investigation. 

Still more important, however, is the general 
fact that even with no limitations of time or 
money analysis can never treat all the consider- 
ation* that may be relevant. Some are intangible. 
For example, how some unilateral U. S. action will 
affect NATO solidarity or whether Congress will 
accept military economies that disrupt cherished 
institutions such as the National Guard or radi- 
cally change the pattern of domestic military 
spending are questions that are hard to handle 
objectively. Considerations of this type can, and 
possibly should, play as important a role in the 
choice of alternative force postures as any ideal- 
ized war outcome calculations. But ways to measure 


these things even approximately don't exist today 
and they must be handled intuitively Other issues 
involve moral judgments: whether national security 

is better served by an increase in the budget for 
defense or for welfare or under what circumstances 
the preservation of an ally is worth the risk of 
general war. The analyst can apply his own judg- 
ment and intuition and that of others to these 
considerations (at least to those of which he is 
aware.'), thus making them part of the study and 
bringing them to the attention of the decision - 
maker . But the man with the responsibility will 
rightly insist on applying his own. 

Measures of effectiveness arc approximate 

In military cost-effectiveness comparisons, 
measures of effectiveness are at best reasonably 
satisfactory approximations for indicating the 
attainment of such vaguely defined objectives as 
deterrence or victory. Sometimes the best that 
can be done is to find measures which point in the 
right direction. Consider deterrence, for instance 
It exists only in the mind--and in the enemy's 
mind at that. We cannot, therefore, measure 
directly the effectiveness of alternatives wc hope 
will lead to deterrence, but must use instead 
approximations such as the potential mortalities 
that v/e might inflict or the roof cover we might 
destroy. Consequently, even if a comparison of 
two force postures indicated that one>could inflict 
50 per cent more casualties on the enemy than the 
other, we could not conclude that this posture 
supplies 50 per cent more deterrence. In fact, 
since it may be important not to look too danger- 
ous, we find arguments that the posture which 
threatens the greatest number of casual ities may 
provide the least deterrence* 

Moreover, we can't be as confident about the 
accuracy of our estimates of effectiveness as we 
are about our cost estimates. It is the opinion 
of analysts who are studying the problem of esti- 
mating potential casualties that these estimates 
could easily be off by factors of three or four. 

In brief, we don't know how to translate a 
capability to create casualties (as perceived by 
the enemy) Into deterrence, we don't know how they 
will compute the casualty-producing capability of 
our forces, and we don't even know how to do it 
ourselves very accurately. 

Don't misunderstand me--the determination of 
even th'* dollar costs of a military action la not 
simple, and to trace out all the resource impli- 
cations of forces and weapons that are as yet only 
concepts la difficult. But once we decide what we 
are costing, we can do fairly well. 

No satisfactory wav to forccaat tiro future, exists 

While it is possible to forecast events to 
come in the sense of mapping out possible futures, 
there is no satisfactory way to predict a single 
future in terms of which we can work out the best 
system or determine an optimum policy. Conse- 
quently, wc must consider a range of possible 
futures or contingencies. In any one of these we 
may be able to designate a preferred course of 
action, but we have no way to determine one for 
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the entire range of possibilities. We can design 
a force structure for a particular war in a 
particular place, but we have no surefire way to 
work out a structure that is good for the entire 
spectrum of future wars in all the places they 
may occur. 

Consequently, defense planning is rich in the 
kind of analysis that tells what damage could be 
done to the United States given a particular enemy 
force structure, but it is poor in the kinds of 
analyses that evaluate how we will actually stand 
in relation to the Soviets in years to come. 

The Virtues 

In view of its defects , is cost-effectiveness 
reliable? If reliability has its colloquial mean- 
ing of being a measure of whether it works or not, 
the answer is yes. This is certainly the opinion 
of the decisionmakers who have made extensive use 
of it. As Charles J. Hitch, then Assistant 
Secretary of Defense, expressed it: 

In a way, it is quite ironic that the very 
people who are so insistent that they want 
the "best and most modern 1 ' in Defense 
hardware , are opposed to the "best and 
most modern" in Defense analysis and 
decision-making techniques. 

The fact that we cannot perform cost-effec- 
tiveness analyses with anything near 100 per cent 
confidence of perfection is no reason to rule out 
their use. The real argument for their use is 
that they provide sounder advice than the alter- 
natives. 

These alternatives have defects too. One 
alternative is pure intuition. It is in no sense 
analytic, since no effort is made to structure the 
problem or to establish cause and effect relation- 
ships and use them to arrive at a solution. The 
process is to learn everything possible about the 
problem, to "live with it," and to let the subcon- 
scious provide the solution. 

Between pure intuition, on the one hand, and 
cost-effectiveness analysis, on the other, there 
are other sources of advice that can, in a sense, 
be considered to employ analysis, although the 
analysis is ordinarily less systematic, explicit, 
and quantitative. One alternative is to turn to 
an expert. His opinion can, In fact, be very 
helpful, if It results from a reasonable and 
impartial examination of the facts, with due 
allowance for uncertainty, and if his assumptions 
and chain of logic are made explicit . For if it 
is explicit, others can use his information to 
lorm their own considered opinion. But an expert, 
particularly an unbiased expert, may be hard to 
find. Another way of handling a problem is to 
turn it over to a committee. Committees, however, 
are much less likely than experts to make their 
reasoning explicit, since their findings are 
usually obtained by bargaining. 

The danger is not that analysis will give the 
wrong advice; it may, of course, but without 
analysis the chances are much higher. And for 
some questions analysis is essential: without 

calculation there is no way to discover how many 


missiles may be needed to destroy a target system, 
or how arms control may affect security. Analysis 
offers an alternative to ‘\nuddling through"; to 
waiting until one can see the problem clearly and 
then attempting to meet the situation. Delay can 
be hazardous; in the world today, there could be 
a crisis or a weapon that could not be handled in 
this way. This is not to say that every aspect of 
such problems can be quantified or that analysis 
is without limitations, but only that it is not 
sensible to formulate policy without careful 
consideration of whatever relevant numbers can be 
discovered. 

Let me draw an analogy between the decision- 
maker using a study team for advice and a medical 
doctor using a clinical laboratory. Suppose, for 
example, our doctor is trying to decide whether 
to send his patient to a surgeon to have his 
stomach resected or to treat him medically for a 
gastric ulcer. The doctor is influenced by: 

1. The technical findings of the laboratory 
crews . Like the decisionmaker, he might or might 
not be able to carry out these investigations 
himself, but it would not be economic for him to 
do so. He depends, therefore, on Laboratory 
reports, some of which will be on cold slips of 
paper without comment or nuance — numbers alone. 
Others from the laboratory might write paragraphs 
or talk to the doctor or bring x-ray plates to 
discuss with him. 

2. Observations or analyses the doctor makes 
himself. Some of these he puts in the form of 
written notes; those he can't write out he retains 
in his head. 

3. Impressions of the risks and possibilities 
of success with various treatments. Some of these 
Impressions are from his experience, others from 
medical reports. 

Finally, like the decisionmaker, the doctor 
must make a judgment baaed on whatever facts or 
analyses he has. This judgment Is the ultimate 
synthesis the doctor makes of the numerical tests, 
the written out but relatively diffused notes , 
the unrecorded conversations with technicians, 
and his own Introspection. It is not a mere 
calculation, but is made on intuitive grounds. 
Sometimes a factor is overriding, but on the 
whole he just doesn't know. He could do more 
analysis, sometimes even risk the patient's life 
in order to guard it- -cal l for a liver puncture 
or other dangerous procedures- -but his Inquiry 
can never be complete. His judgment, like that 
of every decisionmaker, must be made with uncer- 
tainties In mind. 

It Is easy, unfortunately, to exaggerate the 
degree of assistance that analysis can offer a 
policymaker. In almost every case, it can help 
him understand the relevant alternatives and the 
key interactions by providing an estimate of the 
costs, risks, and possible payoffs associated 
with each course of action. In so doing, the 
analysis may sharpen his intuition; it will 
certainly broaden his basis for judgment. This 
can almost always help the decisionmaker make a 
better decision then he would otherwise make, but 
the inherent Limitations mean that a study can 
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seldon demonstrate, beyond alt reasonable doubt, 
tnat a particular course of action is best. 

Now what about quality control? Because coatr 
effectiveness analysis is to a large extent art, 
it is pointless to expect success to follow from 
a set of definite rules. Reliability and quality 
control are not applicable to an art and a high 
degree of accuracy in an absolute sense is mean- 
ingless and impossible. The only way to insure 
that the work is well done and used with its lim- 
itations in mind is through a thorough critique 
by others. For no individual can hope to be 
completely objective. The most we can hope for 
is that they be honest in identifying their bias. 

Th? 

And finally, what of the future? Resistance 
to the use of cost-effectiveness analysis to help 
in broad problems is gradually breaking down. 
Government and industry planning have always 
involved more art than science; what is happening 
is that the art form is changing from an ad hoc . 
seat-of-the-pants approach based on intuition to 
one based on analysis supported bv intuition and 
experience. With this change the computer is 
becoming increasingly s ignif icant--as an automa- 
ton, a process controller, an information proces- 
sor, and a decision aid . Its usefulness in 
serving these ends can be expected to grow. But 
at the same time, it is Important to note that 
even the best computer is no more than a tool to 
expedite analysis. Those advocates who hold that 
decisions can be made today solely by consider- 
ation of computer calculations are not only pre- 
mature in their belief (to say the least), but 
have a basic misunderstanding of how such calcu- 
lations must, in fact, always be used. Even in 
the narrowest decisions, considerations not 
subject to any sort of quantitative analysis can 
always be present. Big decisions, therefore, 
cannot be the automatic consequence of a computer 
program, of cost-effectiveness analysis, or any 
application of mathematical models. 

For broad studies, involving force posture 
and composition or the strategy to achieve foreign 
policy objectives, intuitive, subjective, even 
ad hoc study schemes must continue to be used-- 
but supplemented to an Increasing extent by cost- 
effectiveness analysis. And as ingredients of 
this analysis, along with an increasing use of the 
computer for those problems where it is appropri- 
ate, in recognition of the need for a better 
treatment of the nonquant i f tab le aspects, a greater 
use of techniques for the better employment of 
judgment, Intuition, and experience can be expect- 
ed. These techniques: war gaming, "scenario" 

writing, and the systematic interrogation of 
experts are on the way to becoming an Integral 
part of cost-effectiveness analysis. 

Moreover, the scope will broaden. Cost- 
effectiveness has barely entered the domain of the 
social sciences, where in urban planning, in 
education, in welfare, and in other nonmilitary 
aspects of government we are faced with an abun- 
dance of challenges: how to alleviate the hard- 

ships of social change, how to provide food and 
comfort for the poor, how to improve the social 
Institutions and the values of the affluent, how 


to cope with revolutionary innovations, and so on. 
Cost-effectiveness analysis^ can help with these 
problems as well as those of industry and the 
military. 

C9n£h“4na Rpngrks 

And now to review, A cost-cf fectlvenoss 
analysis is an analytic study designed to assist 
a deciaionmaker identify a preferred choice from 
among possible alternatives. It is characterized 
by a systematic and rational approach, with 
assumptions made explicit, objectives and criteria 
clearly defined, and alternative courses of action 
compared in the light of their possible conse- 
quences. An effort is made to use quantitative 
methods but computers arc not essential. What is 
essential la a model that enables expert intuition 
and judgment to be applied efficiently. The 
method provides its answers by processes that are 
accessible to critical examination, capable of 
duplication by others , and, more or less, readily 
modified as new information becomes available. 

And, in contrast to other aids to decisionmaking, 
which share the same limitations, it extracts 
everything possible from scientific methods, and 
its virtues are the virtues of those methods. At 
its narrowest, cost-effectiveness analysis offers 
a way to choose the numerical quantities related 
to a weapon system so that they are logically 
consistent with each other, with an assumed objec- 
tive, and with the calculator's expectation of the 
future. At its broadest, it can help guide 
national policy. But, even within the Department 
of Defense, its capabilities have as yet It' be 
fully exploited. 

Bibliography 

1. Hitch, Charles J. , Cost/Effect ivencss . 

Military Operations Research Symposium, 

Washington, D. C, , April 29, 1964. 

2. Helmer, Olaf, Social Technology . The RAND 

Corporation, P-3063, February, 196S. 

3. Quadc , Edward S. (od.), Anal vs is for MUi lary 

Peels ions . Rand-McNally & Co., Chicago, 1964. 


28 




29 



